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FOREWORD 


With  the  use  of  increasingly  sophisticated  automated  systems  on  both  the 
current  and  future  battlefields,  the  need  for  efficient,  highly  accurate, 
user-friendly  soldier-machine  interfaces  becomes  critical .  Research  on  un¬ 
derstanding  the  requirements  and  abilities  of  both  the  operator/soldier  and 
the  system  needs  to  focus  on  maximizing  the  match  between  the  soldier  and 
the  machine.  General  research  areas  designed  to  address  this  problem  include 
the  study  of  human  abilities,  computer  capabilities ,  environmental  influences, 
and  specific  task  requirements. 

One  area  of  special  interest  is  the  exploration  of  research  methods 
which  can  be  used  in  the  support  and  enhancement  of  the  decision-making  pro¬ 
cess  within  command,  control,  and  intelligence  centers.  This  technical  re¬ 
port  provides  a  description  of  and  selected  references  for  nine  research 
methods  that  can  be  used  to  examine  cognitive  and  decision-making  processes. 

In  general,  these  methods  can  be  applied  in  the  natural  task  setting  with  a 
minimum  of  interference  with  task  performance.  The  methods  are  classified 
under  three  categories:  process-tracing  methods,  applied  research  methods, 
and  field  research  methods.  While  most  have  not  been  used  previously  to 
study  cognition,  their  imaginative  adaptation  to  this  problem  is  indicated. 
Some  preliminary  suggestions  are  presented  to  help  other  researchers  begin 
to  use  cognitive  procedures  to  understand  operational  decision  making. 


NON LAB ORATORY  TECHNIQUES  FOR  THE  STUDY  OF  COGNITIVE  AND  DECISION-MAKING 
PROCESSES:  A  DESCRIPTION  AND  SELECTED  BIBLIOGRAPHY 


EXECUTIVE  SUMMARY 


Requirement: 

To  summarize  research  methods  that  can  be  used  to  identify  the  cognitive 
factors  that  impact  on  task  performance  when  traditional  laboratory  tech¬ 
niques  would  disrupt  or  modify  performance. 


Approach : 

Literature  from  several  disciplines  was  searched  and  reviewed  for  non¬ 
laboratory  methods  of  potential  value  in  the  investigation  of  cognitive  and 
decision-making  processes.  Methods  of  interest  were  selected  and  individu¬ 
ally  analyzed  for  adaptability  to  the  study  of  cognition. 


Product: 

This  report  summarizes  nine  methods  classified  under  three  main  cate¬ 
gories:  process  tracing,  applied  research,  and  field  research.  Each  method 
is  described  in  terms  of  definition,  history,  procedure,  benefits,  and  limi¬ 
tations,  as  well  as  general  applications.  A  list  of  major  references  is 
made  for  adapting  and  utilizing  each  method  in  the  study  of  cognitive  and 
decision-making  factors. 


Utilization: 

Using  traditional  experimental  techniques  to  test  perceptual,  memory, 
information  processing,  and  decision-making  abilities  is  unsatisfactory'  since 
these  processes  are  likely  to  change  or  interact  during  task  performance 
under  laboratory  conditions.  The  methods  summarized  in  this  report  generally 
can  be  applied  in  the  natural  task  setting  with  minimal  obstruction  or  dis¬ 
ruption  of  task  performance.  While  most  of  these  methods  have  not  been  used 
to  study  cognitive  processes,  all  of  them  shew  some  potential  for  this  use. 
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NONLABORATORY  TECHNIQUES  FOR  THE  STUDY  OF  COGNITIVE  AND  DECISION-MAKING 
PROCESSES:  A  DESCRIPTION  AND  SELECTED  BIBLIOGRAPHY 


INTRODUCTION 

The  purpose  of  this  report  is  to  summarize  research  methods  that  have  po¬ 
tential  for  revealing  the  cognitive  and  decision-making  processes  underlying 
task  performance  when  traditional  laboratory  techniques  cannot  be  used.  The 
emphasis  is  on  methods  that  can  be  applied  in  the  natural  task  setting  with  a 
minimum  of  interference  with  and  disruption  of  the  task  performance.  It  is 
assumed  that  a  definition  already  exists  of  the  task  and  the  environmental 
variables  obtained  by  a  task,  job,  or  systems  analysis.  Yet  to  be  defined  are 
the  cognitive  (thinking,  perception,  memory)  and  decision-making  variables 
necessary  for  task  performance. 

This  search  of  methods  resulted  from  frustration  with  identifying  the 
cognitive  factors  that  affect  task  performance  using  more  conventional  mea¬ 
sures  of  cognitive  processing  such  as  choice,  problem-solving  efficiency, 
reaction  time,  and  recall  and  recognition  performance.  Testing  the  percep¬ 
tual,  memory,  information-processing ,  and  decision-making  abilities  in  labo¬ 
ratory  experiments  is  unsatisfactory  because  these  processes  may  well  change, 
interact,  or  otherwise  metamorphose  during  task  performance  in  the  field 
(Ebbesen  &  Konecni ,  1975,  p.  26). 

While  most  of  the  methods  summarized  in  this  report  have  not  been  de¬ 
signed  or  used  to  study  the  covert  cognitive  processes  underlying  task  per¬ 
formance,  all  show  promise  of  yielding  at  least  some  insights.  Clearly,  an 
innovative  cognitive  or  decision-making  psychologist  will  have  to  adapt  and 
implement  these  procedures. 

This  report  is  organized  into  four  major  sections:  (1)  Overview  of 
Methods,  (2)  Process-Tracing  Methods,  (3)  Applied  Methods,  and  (4)  Field  Re¬ 
search  Methods.  The  Overview  section  summarizes  all  methods  and  their  simi¬ 
larities  and  differences  on  eight  major  variables.  The  three  following  sec¬ 
tions  contain  descriptions  of  each  method  based  on  definition  of  method, 
history,  procedure,  benefits  and  limitations,  general  applications ,  and  sug¬ 
gested  cognitive  applications.  In  addition,  a  list  of  major  references  and 
selected  annotations  is  included  for  each  method. 


OVERVIEW  OF  METHODS 

Eight  major  characteristics  of  each  method  are  summarized  in  Table  1. 
Since  the  emphasis  is  on  data  collection  in  the  field,  most  of  the  charac¬ 
teristics  were  chosen  to  compare  the  bias  introduced  by  each  method  through 
alteration  of  the  natural  task  context  or  intervention  in  task  performance. 
The  methods  are  organized  into  three  categories:  process  tracing,  applied 
research,  and  field  research.  The  following  is  a  description  and  explanation 
of  each  of  the  eight  characteristics: 

1.  Source  of  the  Data — who  or  what  supplies  the  data  of  research 
interest . 


I 


Table  1 


Summary  of  Methods  Compared  on  Eight  Major  Characteristics 


Characteristics  _  ____  _  _ 

Relative 
techni ca 1 


Method 

Source  ol 
the  data 

Nature  *  it  the  data 

Scale  ol 

measure¬ 
ment  ol 

the  data 

exper  1 1  si 
requi red 
by  the 
methoda 

PROCESS  TRAC INC 

1,  verbal  protocols 

Subject 

Verbatim  verbal  res|x>nsc 

Nominal 

2 

2.  Information 
acquisition 

Sub  iect 

Objective  record  of  be¬ 
havioral  responses 

Interval 

Ratio 

2 

APPLIED 

1.  Critical  incident 

Subject , 
observers , 
coworkers , 
etc. 

Subjects'  written 
narratives 

Nominal 

1 

2.  Delphi 

Subject 

Subjects'  opinions 

Nominal 

2 

3.  Operations 
research 

Model  of  the 
system 

Values  of  the  model  that 
optimize  performance 

Ratio 

3 

FIELD  RESEARCH 

1.  Interview 

Researcher 

Subjects'  verbal  responses 
to  questions,  recorded  by 
the  researcher 

Nomi na 1 

1 

2.  Questionnaire 

Subject 

Subjects'  written  responses 
to  questions 

Nominal 

Ordinal 

Interval 

1 

3.  Observation 

Researcher 

Verbal  or  behavioral 
responses  recorded  by  the 
researcher 

Nomi  nal 

Ordinal 

Interval 

Ratio 

1 

4.  Unobtrusive 

measures 
a.  Physical 
traces 

Physical 

Material 

Measures,  weights, 
counts,  etc. 

Interval 

Ratio 

1 

b.  Archives 

Public  & 

private 

records 

Records  or  documents 

Nominal 

1 

c.  Simple 

observation 

Researcher 

Verbal  or  behavioral 
responses  recorded  by 
the  researcher 

Nominal 

Ordinal 

Interval 

Ratio 

1 

d.  Hardware 

Resea rcher 

Recordings  on  tape,  film, 
etc.  of  verbal  or 
behavioral  responses 

Nominal 

Ordinal 

Interval 

Ratio 

1 

The  scale  is  meaningful  only  in  comparing  these  methods;  it  does  not  reflect  any 
absolute  measurement.  The  ^cair-  is  defined  as:  1  =  little  or  none,  2  =  some, 

)  --  a  large  amount. 


Table  1  (Continued) 


Character ist ics 


Method 

Suit iib 1 1  l  ty 
of  method 
for  ex¬ 
ploratory 
research 

Reactivity 
of  the 
method 

Source 

of  bias 

Structuring 
of  the  setting 
required  by 
the  method 

PROCESS  TRACING 

I.  Verbal  protocols 

Yes 

Reactive 

Subject  tk 
researcher 

Could  be 

structured 

2.  Information 
acquisition 

Yes 

Reactive 

Subject 

Yes 

APPLIED 

1.  Critical  incident 

Yes 

Could  beb 
reactive 

Subject 

No 

2.  Delphi 

Yes 

Reactive 

Subject  & 
researcher 

Yes 

3.  Operations 
research 

Yes 

Could  be 
reactive 

Researcher 

Yes 

FIELD  RESEARCH 

1.  Interview 

Yes 

Reactive 

Subject  & 
researcher 

Yes 

2.  Questionnaire 

NO 

Reactive 

Subject 

Yes 

3.  Observation 

Yes 

Reactive 

Subject  & 
researcher 

Could  be 
structured 

4.  Unobtrusive 

measures 
a.  Physical 
traces 

Yes 

Non re active 

Researcher 

No 

b.  Archives 

Yes 

Non reactive 

Author 
(subject)  & 
researcher 

No* 

c.  Simple 

observation 

Yes 

Non reactive 

Researcher 

Could  be 
structured 

d.  Hardware 

Yes 

Non reactive 

Researcher 

Could  be 

structured 

^ "Could  be"  indicates 
dependent  on  the  goal 

that  the  decision 
s  of  the  particular 

is  left  to  the 
study . 

discretion  of 

the  researcher, 

2.  Nature  of  the  Data — the  type  of  data  that  are  collected  and 
analyzed. 

3.  Measurement  Scales  of  the  Data — (a)  nominal:  data  that  are  cate¬ 
gorized  on  the  basis  of  group  membership;  (b)  ordinal:  data  that 
are  ranked,  but  differences  among  the  ranks  are  not  necessarily 
behaviorally  or  numerically  equivalent;  (c)  interval:  data  that 
are  ranked  on  an  equal-interval  scale  that  has  no  absolute  zero; 

(d)  ratio:  data  that  are  ranked  on  an  equal-interval  scale  that 
does  have  an  absolute  zero. 

4.  Relative  Technical  Expertise  Required  by  the  Method--while  skill 
is  required  to  perform  all  of  the  procedures  effectively,  in  this 
instance  technical  refers  to  sophisticated  mathematical  or  inter¬ 
pretive  techniques  or  to  complex  equipment. 

5.  Suitability  of  Method  for  Exploratory  Research — the  potential  a 
plicability  of  the  method  to  the  study  of  complex  or  less  well 
defined  real-world  behavior,  and  to  its  utility  for  providing 
rich  source  of  data  from  which  hypotheses  can  be  generated. 

6.  Reactivity  of  the  Method — bias  or  change  in  behavior  resulting 
from  the  subject's  awareness  of  being  observed  or  tested. 

7.  Source  of  Bias — bias  introduced  in  the  data  by  either  the  subject 
or  the  researcher,  or  both,  and  resulting  from  such  variables  as 
memory  limitations,  selective  recording  of  responses,  or  selective 
encoding  of  data. 

8.  Structuring  of  the  Setting  Required  by  the  Method — whether  the 
data  can  be  collected  from  an  ongoing  task  or  account  of  past  be¬ 
havior,  or  whether  the  task  environment  must  be  structured  to 
elicit  from  the  subject  the  behaviors  of  research  interest. 


PROCESS-TRACING  METHODS 

Traditionally,  in  decision-making  research  the  focus  of  investigation 
has  been  observable  input-output  relationships  rather  than  the  cognitive  pro¬ 
cessing  of  information  that  precedes  a  decision.  In  part  this  has  been  be¬ 
cause  of  the  lack  of  an  adequate  methodology.  Those  investigators  who  have 
tested  the  predecisional  process  itself  have  used  analyses  based  on  algebraic 
models.  For  example,  under  what  set  of  conditions  is  information  cognitively 
integrated  by  averaging?  There  has  been  some  discussion,  however,  about  the 
validity  of  this  model-fitting  approach.  (See  Tversky,  1969,  and  Yntema  & 
Torgerson,  1961,  for  drawbacks,  and  Anderson,  1981,  for  an  alternative  method 
for  testing  algebraic  models.) 

Process -tracing  methodology  offers  another  approach  to  the  investigation 
of  the  cognitive  analysis  and  evaluation  of  information.  This  methodology  is 
derived  from  information-processing  theory  and,  accordingly,  is  designed  to 
identify  the  series  of  cognitive  operations  through  which  information  pro¬ 
gresses  in  the  formulation  of  a  decision.  The  series  of  cognitive  operations 
is  inferred  from  data  such  as  the  order  of  the  information  searched,  what 


information  is  examined  and  what  is  ignored,  the  amount  of  time  each  item  is 
examined,  and  the  time  between  information  requests. 


While  process-tracing  methodology  provides  detailed,  sequentially  ordered 
observations  useful  in  developing  information-processing  models  of  individual 
decision  behavior,  it  docs  have  some  drawbacks.  More  time  and  effort  are  re¬ 
quired  to  conduct  research  using  this  method  and,  in  particular,  to  analyze 
the  resulting  data.  Because  standard  summary  statistics  for  data  collected 
by  process- tracing  studies  are  not  well  developed,  the  researcher  may  have  to 
present  the  results  in  great  detail.  Nevertheless,  some  researchers  consider 
process  tracing  a  valuable  adjunct  to  traditional  input-output  methods  and  a 
useful  tool  in  exploring  real-world  decision  making  (Payne,  Braunstein,  & 
Carroll,  1978). 

Two  kinds  of  process-tracing  techniques  will  be  considered:  verbal  pro¬ 
tocol  analysis,  and  the  analysis  of  information  acquisition  behavior. 
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Verbal  Protocol  Analysis 

Definition.  Verbal  protocols  are  continuous  verbal  reports  supplied  by 
a  subject  during  the  performance  of  a  task.  Verbal  protocols  provide  sequen¬ 
tial,  time-ordered  information  about  knowledge  or  operations  associated  with 
particular  responses. 

History .  The  analysis  of  verbal  protocols  to  obtain  psychological  data 
is  not  a  new  idea.  Introspection  was  commonly  used  around  the  turn  of  the 
century,  and  for  researchers  such  as  Wundt  and  Titchener,  it  was  the  preferred 
method  of  psychological  investigation.  With  the  advent  of  behaviorism,  in¬ 
trospection  was  discouraged  on  the  basis  that  it  was  not  scientific,  verifi¬ 
able,  or  repeatable.  Only  recently,  with  the  availability  of  computers  for 
processing  complex  data  and  the  pioneering  efforts  of  Newell  and  Simon  (1972)  , 
has  the  use  of  verbal  protocols  reemerged. 

It  is  important  to  distinguish  between  verbal  protocol  analysis  and  in¬ 
trospection.  First,  highly  trained  subjects,  and  even  researchers,  have  been 
used  in  introspective  studies,  whereas  subjects  naive  to  the  constructs  of 
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research  interest  are  used  to  provide  protocol  data.  Second,  subjects  using 
introspection  theorize  about  the  causes  and  consequences  ol  theii  behavior, 
in  contrast  to  protocol  analysis,  where  subjects  are  asked  to  report  only 
their  intention  and  state  of  knowledge  at  that  moment.  Finally,  verbal  pro¬ 
tocols  are  collected  during  the  performance  of  a  task  rather  than  retrospec¬ 
tively  through  interviews  or  questionnaires. 

Procedure .  The  subject  is  asked  to  think  aloud  during  the  performance 
of  a  task.  The  subject  should  verbalize  and  record  all  thoughts,  not  only 
those  judged  to  be  of  interest  to  the  researcher.  The  data  are  the  subject's 
verbatim  responses. 

Two  special  procedures  encourage  the  subject  to  verbalize:  (1)  combining 
a  verbal  protocol  technique  with  an  information  search  procedure  (defined  in 
the  next  section)  and  (2)  asking  two  individuals  to  make  a  joint  decision  (Payne 
et  al. ,  1978) . 

Analysis .  Payne  et  al.  (1978)  summarized  procedures  for  the  analysis  of 
protocol  data.  Following  complete  transcription  of  the  verbal  report,  the 
first  step  in  the  analysis  is  to  break  up  the  protocols  into  short  phrases, 
each  corresponding  to  the  assessment  of  or  reference  to  a  single  task.  Hie 
next  step,  encoding  the  phrases  into  formal  categories,  is  a  controversial 
one.  Newell  and  Simon  (1972)  advocate  dealing  with  the  data  in  the  form  of 
short  phrases  if  possible,  because  encoding  may  result  in  a  substantial  loss 
of  information.  However,  rigorous  analysis  of  the  data  will  probably  require 
some  form  of  encoding.  Newell  and  Simon  (1972)  have  developed  one  encoding 
method  known  as  the  problem  behavior  graph  (PBG) ,  which  represents  the  se¬ 
quence  of  operations  and  acquired  knowledge  states  of  a  subject  in  the  pro¬ 
cess  of  reaching  a  decision.  A  node,  designated  by  a  rectangle,  represents 
a  state  of  knowledge.  A  horizontal  arrow  to  the  right  between  two  nodes 
represents  the  application  of  an  operation  to  generate  a  new  state  of  knowl¬ 
edge.  Examples  of  operations  are  search,  comparison,  evaluation,  goal  state¬ 
ment,  and  choice.  (See  Payne  et  al.,  1978,  for  a  detailed  example.) 

Benefits  and  Limitations.  A  high  temporal  density  of  sequential  data  is 
characteristic  of  process-tracing  techniques  in  general.  These  data  are  use¬ 
ful  for  making  inferences  about  decision  rules  and  information-processing 
models  of  decision  behavior.  Of  particular  importance  is  the  value  of  this 
methodology  to  the  study  of  real-world  decision  making.  Although  it  is  clear 
that  the  think-aloud  procedure  is  not  unobtrusive,  the  limited  information 
about  this  procedure  indicates  that  while  verbal  protocols  slow  the  decision 
process,  they  do  not  alter  it  in  a  fundamental  way  (Carroll  &  Payne,  1977). 

Another  question  associated  with  the  use  of  verbal  protocols  is  one  of 
coding  reliability.  Empirical  investigations  of  reliability  have  produced 
conflicting  results,  but  Payne  et  al .  (1978)  concur  with  Simon  and  Newell 
(1974)  that  even  though  some  subjectivity  does  enter  into  the  process,  it  is 
not  very  great.  These  are  two  possible  approaches  to  increased  reliability. 

The  researcher  can  validate  interpretations  of  the  data  with  other  measures, 
such  as  information-search  procedures  or  measurement  of  response  time  (such 
as  amount  of  time  to  make  a  choice) ;  and  the  process  can  be  computerized 
(Waterman  &  Newell,  1973). 
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Applications .  Verbal  protocol  analysis  has  a  number  ol  possible  appli¬ 
cations  to  decision  research.  First,  it  can  be  used  for  exploratory  research 
of  complex,  real-world  decision  behavior.  This  methodology  proves  to  be  in¬ 
creasingly  valuable  as  the  decision  task  becomes  more  complex  and  less  well 
defined.  Furthermore,  coded  protocols  may  serve  as  the  basis  for  descriptive 
statistics,  such  as  the  percentage  of  protocols  falling  in  particular  cate¬ 
gories.  Second,  protocols  can  be  used  to  supplement  and  validate  data  col¬ 
lected  by  other  methods.  Third,  protocols  can  be  used  to  test  specific  hy¬ 
potheses  since  the  data  collected  allow  categorization  and  some  quantification. 
Finally,  protocols  can  be  used  to  develop  and  test  computer  models  of  decision 
behavior  (Payne  et  al . ,  1978). 
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Analysis  of  Information-Acquisition  Behavior 

Definition.  Monitoring  information-acquisition  behavior  involves  pre¬ 
senting  a  decision  task  in  such  a  way  that  the  subject's  selection  and  exami¬ 
nation  of  information  can  be  easily  monitored.  Tins  method  provides  data  on 
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the  information  sought,  the  order  of  search,  the  depth  of  search,  and  the  du¬ 
ration  of  examination  for  each  piece  of  information. 

Accordinq  to  Svenson  (19  79)  ,  three  assumptions  are  associated  with  this 
procedure.  First,  a  piece  of  information  is  assumed  to  be  processed  cogni¬ 
tively  while  the  subject  examines  it.  Second,  directed  attention  is  assumed 
to  reflect  the  search  for  information  necessary  to  comply  with  a  predetermined 
decision  strategy.  Third,  a  longer  fixation  on  a  single  item  is  assumed  to 
represent  a  more  complex  cognitive  process. 

Procedure .  TVo  general  methods  have  been  used  for  monitoring  information- 
acquisition  behavior.  The  first  method,  eye  movement  recording,  has  been  used 
for  a  number  of  years  in  psychological  research  and  has  usually  emphasized  the 
length  of  time  (seconds,  milliseconds)  the  eyes  are  actually  fixated  on  dif¬ 
ferent  types  or  sources  of  information.  In  decision-making  research,  however, 
the  emphasis  is  placed  on  the  sequence  of  fixations;  that  is,  the  serial  order 
of  fixation  on  the  various  sources  of  information.  Thus,  for  fixation  dura¬ 
tion  the  data  collected  are  the  actual  fixation  times,  whereas  for  fixation 
sequence,  the  data  are  strings  of  rank-ordered  sources  of  information. 

Although  eye  movement  recordings  provide  data  that  are  detailed  and  hard 
to  misrepresent,  some  technical  problems  are  associated  with  their  use.  The 
apparatus  is  expensive,  cumbersome,  and  uncomfortable  for  the  subject.  In  ad¬ 
dition,  only  a  limited  amount  of  information  can  be  displayed,  particularly 
on  a  CRT,  if  eye  movement  is  to  be  precisely  detected.  Recently,  the  use  of 
videotapes  has  eliminated  some  of  these  problems  and  shows  potential  value  as 
a  recording  device  in  more  realistic  environments  (Russo,  1978) . 

The  second  method  of  information-acquisition  analysis  involves  the  sub¬ 
ject's  explicit  search  for  hidden  information.  Three  techniques  have  been 
used  to  collect  search  data.  The  first,  information  boards,  consists  of  a 
matrix  of  envelopes  attached  to  a  cardboard  sheet  (Payne,  1976).  The  subject 
must  pull  a  card  out  of  the  envelope  to  obtain  information.  Another  tech¬ 
nique  utilizes  a  random-access  slide  projector  (Carroll  &  Payne,  1977)  .  The 
subject  must  request  a  specific  category  by  number  from  a  list;  information 
in  each  category  is  available  on  one  slide.  A  third  technique,  involving 
computer-controlled  retrieval,  requires  that  the  subject  press  keys  to  dis¬ 
play  information  on  a  CRT  screen  (Payne  &  Braunstein,  1978) . 

Analysis .  Analysis  of  information-acquisition  behavior  is  more  straight¬ 
forward  than  analysis  of  verbal  protocols,  primarily  because  the  procedure 
focuses  on  the  subject's  external,  objective  search  behavior.  Thus,  the  data 
collected  can  be  quantified  and  standard  statistical  comparisons  can  be  easily 
made.  In  addition,  more  subjects  are  used  in  information  search  studies  than 
in  verbal  protocol  studies,  providing  the  potential  for  revealing  behavioral 
regularities  more  rapidly  and  reliably. 

The  procedure  usually  involves  an  examination  of  the  content,  amount  or 
depth  of  search,  and  the  pattern  of  the  search.  For  example,  one  measure  of 
particular  interest  in  decision-making  research  is  whether  a  subject  searches 
across  all  the  alternatives  within  each  attribute  or  across  all  the  attri¬ 
butes  within  each  alternative. 


Benefits  and  Linu  t at  ions .  UnlLke  verbal  protocol  analysis,  information- 
acquisition  procedures  provide  data  only  on  overt  behavioral  responses.  They 
do  not  provide  information  on  whether  or  not  examined  items  are  actually  beinq 
processed  or  information  stored  in  memory  is  being  used  in  reaching  a  decision. 
Furthermore,  information-search  procedures  usually  require  the  task  to  be  more 
structured  and  less  complex.  Although  more  structure  leads  to  more  easily  ana- 
lyzable  results,  it  also  reduces  the  number  and  types  of  realistic  environments 
in  which  it  can  be  used  (Payne,  1980,  p.  111). 

Applications .  Information-acquisition  procedures  may  be  used  to  test 
specific  hypotheses  about  the  decision  process.  For  example,  a  researcher 
can  monitor  the  information  used  at  a  computer  terminal  to  test  hypotheses 
about  the  types  of  data  that  are  used,  the  order  in  which  the  data  are  used, 
the  influence  of  organizing  the  data  base  in  different  ways,  and  the  point  at 
which  the  search  for  information  is  completed  and  a  decision  is  made.  Another 
important  use  is  the  validation  of  other  methods  in  a  multimethod  approach. 
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Behavior  and  Human  Performance,  23,  86-112.  Presented  in  this  paper  is  a 
discussion  of  two  process-tracing  techniques:  verbal  protocol  analysis 
and  information-acquisition  procedures.  A  number  of  assumptions  associ¬ 
ated  with  these  techniques  are  delineated.  Studies  of  decision  making 
that  use  process-tracing  methodology  are  reviewed.  The  review  indicates 
that  an  information-search  pattern  is  a  function  of  the  number  of  alter¬ 
natives  and  attributes,  the  presentation  format  of  information,  the  use 
of  different  rules  during  different  stages  of  the  process,  the  order  of 
decision  rules,  and  absolute  versus  comparative  evaluation  statements 
about  attributes.  Research  in  each  of  these  areas  is  discussed.  One 


important  finding  is  that  subjects  make  decisions  without  completing  a 
search  of  the  data.  Therefore,  many  of  the  algebraic  decision  models 
which  assume  a  completed  data  search  are  inadequate.  A  representation 
system  for  describing  decision  rules  is  introduced  and  some  examples  of 
these  decision  rules  are  given. 


Cognitive  Applications 

The  process-tracing  methods  have  been  designed  specifically  to  investi¬ 
gate  cognitive  processes  and  to  trace  information  processing.  These  methods 
are  cased  on  the  rationale  that  to  answer  questions  about  the  psychological 
processes  involved  in  cognition  and  decision  making  (learning,  problem  solv¬ 
ing,  thinking,  memory,  and  perception) ,  data  should  be  collected  as  frequently 
as  possible  during  the  subject's  actual  psychological  processing  of  the  task. 
The  methods  are  particularly  useful  for  studying  questions  like  what,  how, 
and  how  much  information  is  used  to  make  a  choice  or  judgment;  whether  the 
information  stored  in  memory  is  used;  what  are  the  effects  of  the  presenta¬ 
tion  format  of  information;  and  what  are  the  effects  of  different  sources  of 
data . 


One  important  application  of  this  method  is  the  development  of  computer 
information-processing  models  from  data  generated  by  process-tracing  studies. 
These  models  simulate  the  flow  of  information  through  various  cognitive  states 
and  operations.  Computer  models  afford  the  advantage  of  precision  by  allowing 
testable  predictions  about  behavior.  In  addition,  through  analysis  of  the  in¬ 
fluence  of  particular  variables  on  the  model's  components,  computer  models  al¬ 
low  a  determination  of  the  locus  of  various  effects. 

There  are  at  least  three  other  application  domains  for  process-tracing 
methods:  (1)  In  real-world  settings,  process-tracing  methods  can  be  used  to 

test  and  extend  the  results  of  more  traditional  laboratory  input-output  and 
model-fitting  studies;  (2)  for  exploratory  research  in  real  settings,  these 
methods  can  be  used  to  search  for  regularities  in  information  processing  and 
to  provide  the  experimenter  with  a  basis  for  hypothesis  generation;  and 
(3)  in  both  laboratory  and  natural  settings,  process-tracing  methods  can  be 
used  to  test  specific  hypotheses  pertaining  to,  for  example,  the  type  of  data 
sought  by  a  decision  maker  or  the  influence  of  different  data  base  models. 


APPLIED  RESEARCH  METHODS 

Applied  research  methods  are  typically  used  when  the  focus  of  an  in¬ 
vestigation  is  on  concrete  phenomena  involved  in  practical  problems.  This 
is  distinguished  from  a  more  abstract  or  theoretical  focus  common  to  experi¬ 
mental  investigations. 

While  the  applied  methods  described  in  this  section  are  widely  used,  they 
are  rarely,  if  ever,  used  to  study  psychological  problems.  The  critical  in¬ 
cident  technique  comes  from  the  area  of  industrial  and  organizational  psy¬ 
chology  and  is  used  primarily  to  develop  criteria  for  evaluation  and  placement 
of  personnel,  as  well  as  for  job  and  systems  analysis.  Furthermore,  this 
technique  is  frequently  employed  in  human  factor  evaluations  of  design  errors 
in  systems.  The  Delphi  method  has  been  used  as  a  forecasting  and  planning 
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tool  tor  very  large-scale  systems  and  issues.  Operations  research  is  typically 
applied  to  management  and  control  situations  and  recently  to  human  factors 
problems  of  work  space  and  display  design. 


This  section  includes  separate  discussions  of  each  of  these  three  applied 
research  methods  and  concludes  with  suggestions  for  their  application  in  cog¬ 
nitive  and  decision-making  research. 


Critical  Incident  Technique 

Definition .  The  critical  incident  technique  is  a  procedure  in  which  in¬ 
cidents  of  outstanding  behavior  (both  good  and  bad)  are  recorded  in  the  form 
of  anecdotes  by  qualified  observers  (subordinates,  superiors,  or  coworkers). 

It  is  useful  for  sampling  the  many  behaviors  that  make  up  a  job  or  activity 
and  for  defining  the  actual  behaviors  that  characterize  effective  of  ineffec¬ 
tive  performance. 

History.  The  technique  was  developed  by  Flanagan  (1954)  and  was  first 
applied  to  the  problem  of  vertigo  experienced  by  pilots  in  the  Air  Force  dur¬ 
ing  World  War  II.  This  study  led  to  a  number  of  recommendations  for  changes 
in  cockpit  and  instrument  design  and  in  training  procedures. 

Procedure.  Anecdotes  of  critical  behavior  are  collected  that  describe 
the  events  leading  up  to  the  incident,  the  outstanding  behavior  performed  in 
response  to  the  incident,  and  the  perceived  consequences  of  the  behavior. 

The  anecdotes  are  then  abstracted  and  grouped  into  unambiguous  and  meaning¬ 
ful  behavioral  categories.  From  the  categorical  data,  a  comprehensive  check¬ 
list  or  questionnaire  is  derived  that  empirically  defines  critical  job  be¬ 
haviors  and  can  then  serve  as  a  research  instrument  in  further  job  analysis. 
(See  Campbell,  Dunnette,  Lawler,  &  Weick,  1970,  for  a  detailed  example.) 

Benefits  and  Limitations.  The  critical  incident  technique  defines  an 
activity  in  terms  of  actual  rather  than  ideal  or  desirable  behavior.  Anec¬ 
dotes  are  collected  from  the  persons  best  suited  to  provide  this  information — 
direct  observers  of  the  behavior.  Further,  the  questionnaire  is  not  based  on 
a  global  job-success  dimension.  Rather,  it  is  based  on  data  collected  from  a 
specific  group  or  organization  and  is,  therefore,  tailor-made  for  that  group. 
To  ensure  sufficient  coverage  of  the  domain  of  required  behaviors,  hundreds 
or  even  thousands  of  incidents  must  be  collected.  This  may  or  may  not  be  a 
problem,  depending  on  the  population  being  sampled.  The  investigator  should 
also  be  aware  that  oversimplification  of  behavioral  dimensions  will  result  in 
loss  of  impact  and  usefulness  of  the  questionnaire. 

One  disadvantage  of  this  technique  is  the  investigator's  inability  to 
assess  the  relative  frequency  of  particular  behaviors.  Flanagan  (1954)  re¬ 
ported  that  the  total  number  of  incidents  recalled  is  a  function  of  time  and 
that  the  recall  over  time  of  certain  types  of  incidents  is  selective. 

Applications .  The  critical  incident  technique  has  been  used  to  develop 
selection  and  classification  measures,  proficiency  measures,  and  training 
programs,  and  to  collect  data  for  job  and  systems  analysis.  In  addition,  it 
is  frequently  used  by  human  factors  specialists  to  investigate  accident  data 
in  order  to  analyze  design  errors  in  a  system. 


REFERENCES  AND  SELECTED  ANNOTATIONS 

Campbell,  J.  D.  ,  Dunnette ,  M.  D.,  Lawler,  E.  E.,  II,  &,  Weick,  K.  E.,  Jr. 

(1970)  .  Managerial  behaviors,  performance,  and  effectiveness.  New 
York:  McGraw-Hill,  pp.  77-83. 

Chapanis ,  A.  (1959).  Research  techniques  in  human  engineering.  Baltimore: 
Johns  Hopkins  University  Press,  pp.  88-92. 

Flanagan,  J.  C.  (1954,  July).  The  critical  incident  technique.  Psychologi¬ 
cal  Bulletin,  51 ,  327-358.  The  development  and  fundamental  principles 
of  the  critical  incident  technique  are  described  by  the  author  who  de¬ 
veloped  the  technique  during  the  mid-1940s.  A  review  of  the  results  of 
a  number  of  studies  employing  the  procedure  is  presented.  Practical  ap¬ 
plications  of  the  technique  in  the  areas  of  criteria  measures,  profi¬ 
ciency  measures,  selection  and  classification  measures,  operating  pro¬ 
cedures  development,  job  and  equipment  design,  and  in  the  study  of 
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Delphi  Method 

Definition.  The  Delphi  method  is  a  structured  communication  process  that 
allows  individual  assessment  and  revision  of  a  group  judgment  while  individual 
responses  remain  anonymous.  As  a  decision-making  aid,  the  Delphi  method  is 
used  to  obtain  a  reliable  consensus  from  a  group. 

History.  The  Delphi  technique  was  developed  more  than  two  decades  ago 
by  the  creative  effort  of  an  Air  Force-sponsored  RAND  Corporation  "Project 
Delphi."  It  was  first  utilized  to  gain  a  consensus  from  a  group  of  experts 
on  the  effects  of  a  hypothetical  Soviet  atomic  boirbing  attack  on  the  United 
States  (Dalkey  &  Helmer,  1963) . 

Procedure .  Two  distinct  forms  of  the  Delphi  process  are  in  use  today. 

In  the  "Delphi  Exercise,"  or  conventional  Delphi,  questionnaires  are  sent  to 
a  group  of  participants.  Following  the  return  of  the  questionnaires,  a  moni¬ 
tor  team  consolidates  the  responses  and  develops  a  new  questionnaire  based 
on  these  results.  Through  repeated  questioning  and  controlled  feedback,  in¬ 
dividual  estimates  tend  to  converge.  A  newer  form  called  "Delphi  Conference," 
or  the  real-time  Delphi,  uses  a  computer  that  has  been  programed  to  summarize 
each  round  of  responses . 

Variations.  A  variation  of  Delphi,  the  "Policy  Delphi,"  introduced  by 
Turoff  (1970),  is  used  to  generate  the  strongest  possible  opposing  views  on 
the  potential  resolution  of  a  major  policy  issue,  that  is,  a  policy  for  which 
there  are  no  experts,  only  advocates.  The  Policy  Delphi  departs  from  the 
original  form  of  Delphi  in  that  the  decision  maker  is  interested  in  being 
presented  with  all  options  and  supporting  evidence  rather  than  with  a  formed 
consensus  from  the  group.  This  technique  has  been  of  benefit  as  a  precursor 
to  committee  activity.  Participants  can  propose,  assess,  and  react  to  dif¬ 
fering  viewpoints  without  fear  of  political  or  personal  repercussions.  The 
resulting  viewpoints  can  be  utilized  by  a  small  workable  committee  to 
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formulate  a  policy.  For  example,  the  Policy  Delphi  has  been  used  to  obtain  a 
rank-ordered  list  of  national  priority  areas  that  could  create  major  public 
problems  in  the  near  future. 


Another  variant  of  Delphi,  advocated  by  Delbecq,  involves  four  steps. 
First,  each  participant  makes  an  initial  estimate.  Second,  the  estimates 
are  presented  to  the  group.  Next,  participants  discuss  their  estimates  and 
the  rationales  on  which  they  are  based.  Finally,  each  participant  anonymously 
reaches  a  conclusion.  The  approach  differs  from  the  original  Delphi  technique 
in  that  direct  discussion  replaces  the  feedback  provided  by  a  monitor  team. 

Benefits  and  Limitations.  The  Delphi  method  is  appropriate  when  accurate 
information  is  unavailable  or  too  costly  to  obtain.  It  is  also  useful  when  a 
great  many  subjective  estimates  must  be  made  in  reaching  a  decision  on  a  com¬ 
plex  issue.  Other  advantages  of  this  method  are  that  participants  do  not  have 
to  meet  in  the  same  physical  location,  and  that  independent  thought  is  fos¬ 
tered  because  forceful  members  are  prevented  from  dominating  the  discussion. 
Delphi  should  not  be  used  when  the  issue  under  discussion  has  been  previously 
developed  and  refined;  participants  seldom  build  meaningfully  on  elaborated 
concepts . 

Applications.  Delphi  has  been  used  as  a  communication  system  and  a  tech¬ 
nological  forecasting  tool.  It  has  been  widely  applied  to  government,  indus¬ 
trial,  and  organizational  planning,  and  more  recently  applied  to  medical  con¬ 
cerns.  For  instance,  a  Delphi  was  conducted  to  assess  the  impact  of  nutrition, 
family  income,  and  prenatal  care  on  birth  weight  and  intellectual  development. 
The  results  were  incorporated  into  a  cost-benefit  analysis  of  government  nu¬ 
trition  programs.  In  addition,  the  Delphi  method  has  been  used  to  establish 
priorities  and  isolate  significant  parts  of  a  problem.  For  example,  a  Delphi 
was  conducted  with  computer  systems  designers  in  order  to  gather  and  organize 
their  knowledge  about  important  features  and  characteristics  of  computer  sys¬ 
tems  (Turoff,  Hiltz,  &  Kerr,  1982);  the  points  on  which  the  systems  designers 
disagreed  highlight  design  choices  that  need  empirical  study.  Delphi  also 
has  been  applied  to  the  study  of  historical  events  and  to  the  evaluation  of 
historical  factors  leading  to  the  development  of  a  technology.  In  general, 
Delphi  is  finding  application  in  the  more  complex  problem  areas  facing  so¬ 
ciety:  transportation ,  environment,  education,  and  so  forth  (Lindstone  & 

Turoff,  1975)  . 
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Operations  Research 

Definition .  Operations  research  (OR)  applies  specialized  techniques  to 
problems  that  arise  in  the  management  and  control  of  humans,  machines,  mate¬ 
rial,  and  money  in  their  natural  environments  (Chapanis,  1961).  OR  attempts 
to  aid  administrative  decisions  about  system  performance  by  providing  quanti¬ 
tative  solutions  to  situations  involving  conflicting  goals.  The  essence  of 
the  OR  approach  is  to  construct  a  mathematical  model  of  the  system  and  con¬ 
duct  research  in  the  form  of  simulations  or  mathematical  analyses  to  derive 
a  solution.  A  solution  refers  to  the  values  in  a  model  that  optimize  perfor¬ 
mance  measures.  A  solution  is  derived  by  analyzing  the  relationships  that 
determine  the  consequences  of  decision  choices.  Operations  research  utilizes 
a  number  of  methodologies  in  deriving  solutions.  Among  these  methodologies 
are  linear  programing,  operational  gaming,  inventory  modeling,  and  decision 
theory . 

History ■  Although  its  roots  extend  to  the  19th-century  Industrial  Revo¬ 
lution,  the  term  operations  research  was  coined  during  World  War  II.  OR 
rapidly  advanced  as  a  discipline  during  this  period.  Its  present  form  grew 
out  of  an  attempt  by  the  British  military  to  take  action  against  heavy  German 
air  attack.  Teams  of  scientists  from  a  variety  of  disciplines  were  called  in 
to  aid  the  British  military  executives  in  incorporating  the  then  new  radar 
into  military  strategy.  Due  to  the  success  of  these  teams,  the  United  States, 

Canada,  and  France  demanded  similar  teams  of  scientists  that  usually  were  as¬ 
signed  to  the  military  executive  in  charge  of  operations.  Following  the  war, 
defense  research  in  Britain  was  reduced,  thus  allowing  many  OR  specialists 
to  be  hired  by  industry.  In  the  United  States,  however,  defense  research  was 
increased  and  most  OR  specialists  remained  in  the  military  service.  In  more 
recent  years,  with  the  advent  of  computer  technology,  greatly  increased  com¬ 
petition,  and  the  need  for  innovative  methods  to  collect  and  analyze  data, 
operations  research  applications  in  business  have  grown  (Ackoff  &  Sasieni, 

1968)  . 

Procedure.  According  to  Ackoff  and  Sasieni  (1968) ,  there  are  five  steps 
to  the  OR  approach: 

1.  formulate  the  problem, 

2.  construct  a  mathematical  model  of  the  system, 

3.  derive  a  solution, 

4.  test  the  model  and  evaluate  the  solution,  and 

5.  implement  and  maintain  the  solution  in  the  system. 

To  formulate  a  problem,  the  minimum  amount  of  information  needed  con¬ 
sists  of:  (1)  definition  of  the  decision  maker,  (2)  the  courses  of  action 
available  and  the  choice  preferred,  and  (3)  determination  of  the  controllable 
and  uncontrollable  variables  and  the  restrictions  to  be  placed  on  them. 

Procedures  for  constructing  models  fall  into  one  of  five  patterns,  de¬ 
pending  on  the  complexity  of  the  system  and  the  amount  of  access  the  researcher 
has  to  the  system.  Model  construction  might  be  based  on:  (1)  direct  observa¬ 
tion  of  system  operations;  (2)  an  analogous  system  whose  structure  is  better 
known;  (3)  analysis  of  data  that  describes  the  system  operations;  (4)  experi¬ 
mental  determination  of  relevant  variables;  or  (5)  particularly  for  large- 
scale  conflicts,  an  artificial  reality  characteri zed  as  a  complex  game  in 
which  many  hypotheses  can  be  tested. 
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Numerous  techniques  fot  derivmq  solutions  irom  models  have  been  used  in 
OR.  For  instance,  several  types  of  mathematical  programing  have  been  devel¬ 
oped:  Linear,  nonlinear,  inteqer,  dynamic,  stochastic,  parametric,  and  net¬ 

work  analysis.  They  are  usually  applied  to  problems  of  transportation  routing 
and  allocation  of  resources  to  jobs.  Another  technique,  queuing  theory,  is 
typically  applied  to  problems  of  congestion  (in  traffic  and  telecommunica¬ 
tions)  ,  airline  passenger  checkin,  and  passenger  air  terminal  design.  A  third 
technique,  game  theory,  is  frequently  applied  to  problems  of  competition  and 
conflict,  particularly  of  a  military  or  political  nature.  Decision  theory, 
still  another  technique,  is  applied  to  a  one-person,  two-person,  or  two-group 
bargaining  situation  in  which  optimum  choice  is  made  under  uncertainty.  An 
example  is  the  one-person  situation  in  which  a  judge  must  decide  whether  to 
release  a  defendant  on  bail  (Ackoff  &  Sasieni,  1968;  DeGreene ,  1970). 

Benefits  and  Limitations.  Operations  research  offers  a  number  of  poten¬ 
tial  benefits:  (1)  better  decisions  because  an  OR  model  can  account  for  more 
information  than  the  human  can  process;  (2)  better  coordination  of  decisions, 
for  example,  marketing  decisions  with  manufacturing  capabilities;  and  (3)  bet¬ 
ter  control  over  daily  operations,  to  free  executives  from  supervision  of 
routine  matters  (Wagner,  1969) . 

While  many  operations  research  problems  and  techniques  are  of  potential 
interest  in  the  area  of  psychology,  there  has  been  little  effort  to  create  a 
liaison  between  the  two  fields.  Topmiller  (1968)  reported  conclusions  from 
Leuba's  review  of  250  OR  studies,  which  indicated  that  psychological  variables 
were  so  deeply  incorporated  in  OR  models  that  extrication  and  assessment  of 
the  variables  per  se  were  very  difficult.  The  lack  of  standard  measures  of 
human  performance  and  the  divergence  of  psychological  parameters  also  prevent 
quantitative  comparison.  Moreover,  OR  tends  to  emphasize  theory  over  appli¬ 
cations  and  to  concentrate  more  on  the  model  than  on  the  real  system.  A 
final  important  limitation  is  that  operations  research  requires  more  mathe¬ 
matical  background  than  most  psychologists  possess,  or  want  (DeGreene,  1970)  . 

Applications ■  Operations  research  techniques  are  typically  applied  to 
problems  of  (1)  inventory,  (2)  allocation,  (3)  queuing,  (4)  sequencing  and 
coordination,  (5)  routing  and  transportation,  (6)  replacement,  (7)  competi¬ 
tion  and  conflict,  and  (8)  search.  Human  factors  problems  that  have  been 
investigated  with  OR  methods  are  work  space  design  and  panel  layout,  visual 
sampling  and  display  design,  information  system  data  file  and  data  retrieval, 
organizational  data  flow,  and  personnel  allocation.  For  example,  Carbonell 
et  al.  (1968)  applied  queuing  theory  to  the  problem  of  visual  sampling  of 
aircraft  cockpit  instrumentation.  According  to  their  model,  different  in¬ 
struments  compete,  or  queue,  for  the  pilot's  attention.  The  instruments  that 
are  looked  at,  or  served,  are  those  with  highest  priority.  Priority  is  de¬ 
termined  by  the  probability  of  exceeding  a  threshold  (leading  to  a  catastrophe) 
and  the  cost  of  exceeding  that  threshold  (DeGreene,  1970). 
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Cognitive  Applications 

To  date,  the  methods  discussed  in  this  section  have  not  been  applied  in 
the  study  of  cognitive  and  decision-making  problems.  However,  each  does  show 
some  potential  for  adaptation  and  implementation  in  the  examination  of  these 
areas . 

One  possible  use  of  the  critical  incident  technique  is  to  define  excep¬ 
tionally  effective  or  ineffective  decision  processes.  The  incidents  might 
serve  as  the  basis  for  a  cognitive  job  analysis.  Individuals  such  as  medi¬ 
cal  technicians  or  military  intelligence  analysts,  whose  jobs  consist  largely 
of  cognitive  activity,  would  be  ideal  populations  from  which  to  draw  subjects. 
Two  approaches  might  be  considered  in  the  collection  of  critical  incidents. 

One  is  to  ask  subjects  to  speculate  on  how  they  arrived  at  particular  faulty 
decisions.  In  other  words,  what  factors  or  aspects  relevant  to  the  situation 
were  not  taken  into  account  or  were  incorporated  wrongly  in  the  decision 
process,  leading  to  an  error?  As  an  alternative,  subjects  would  report  cor¬ 
rect  decisions  that  were  made  contrary  to  those  of  most  coworkers,  and  specu¬ 
late  about  their  perception  and  consideration  of  important  factors  or  vari¬ 
ables  that  preceded  those  decisions. 

The  format  of  and  concept  behind  the  Delphi  method  are  well  suited  to 
its  application  in  cognitive  areas.  Delphi  is  at  its  best  when  applied  to 
complex  areas  for  which  many  subjective  estimates  must  be  made  in  reaching  a 
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decision.  Using  tins  method,  a  |mu|  of  expel  t1  . -oulu 
the  subjective  process  involved  m  a  cognitive  problem.  Thf>uqli  rejeate.i 
questioning  and  feedback,  a  consensus  >f  'he  relevant  (actors  and  decision: 
could  be  obtained. 

While  there  appears  to  be  no  direct  way  to  obtain  data  about  cognition 
with  operations  research  techniques,  OK  models  might  be  used  for  testing  the 
implications  of  cognitive  processing.  To  do  this,  a  model  representing  ex¬ 
pected  or  typical  processing  of  a  particular  cognitive  problem  could  be  con¬ 
structed.  When  varying  amounts  of  information,  say  20%  and  50%  of  that  re¬ 
quired,  are  entered  in  the  model,  the  derived  solutions  should  predict 
performance  and  suggest  the  types  of  errors  humans  will  make  under  similar 
conditions.  In  addition,  the  models  could  be  used  to  test  the  consequences 
of  already  identified  human  biases.  For  example,  one  of  the  model's  param¬ 
eters  could  be  the  relative  importance  or  weight  given  to  base  rate  informa¬ 
tion.  Solutions  then  could  be  obtained  for  different  values  of  the  base 
rate  weight,  such  as  0%,  20%,  and  80%. 

FIELD  RESEARCH  METHODS 

Field  research  is  used  to  collect  data  from  persons  acting  out  their 
typical  roles  in  natural  settings  rather  than  in  the  laboratory.  In  compari¬ 
son  to  laboratory  research,  field  research  has  a  number  of  advantages  that 
allow  greater  generality,  applicability,  and  utility  of  knowledge.  In  the 
field  there  are  both  greater  intensity  of  experience  and  variation  of  phe¬ 
nomena  than  can  be  stimulated  in  the  laboratory.  Moreover,  laboratory  con¬ 
ditions  may  not  provide  sufficient  frequency  and  duration  of  stimulation  to 
cross  a  response  threshold  and  trigger  an  effect.  Three  contextual  factors 
that  are  typically  destroyed  in  the  laboratory  in  an  attempt  to  achieve  con¬ 
trolled  conditions  are  (1)  natural  time  constraints  such  as  the  life  span  of 
certain  phenomena,  (2)  natural  units  of  behavior  that  occur  in  conjunction 
with  particular  environments,  and  (3)  environmental  complexity.  Two  addi¬ 
tional  factors,  representativeness  of  treatments  and  setting  effects,  while 
not  completely  absent  from  the  laboratory,  are  stronger  cr  more  dynamic  in 
the  field.  Field  research,  however,  is  not  without  its  pitfalls.  Two  spe¬ 
cial  problems  are  the  causal  ambiguity  associated  with  uncontrolled  condi¬ 
tions  and  the  greater  expense  of  field  work  in  terms  of  both  time  and  money 
(Bouchard,  1976) . 

The  major  field  methods  of  data  collection  are  interviews,  question¬ 
naires,  observation,  and  unobtrusive  measures.  Each  method  will  be  consid¬ 
ered  separately,  followed  by  a  discussion  of  potential  applications  to  cog¬ 
nitive  and  decision-making  research. 


Interview 


Definition.  An  interview  is  a  conversation  focused  by  an  investiqator 
to  obtain  self-report  data  from  a  respondent. 

History  and  Applications.  'Hie  interview  has  a  long  and  extensive  history 
in  a  wide  variety  of  fields  and  has  become  a  fixture  in  such  areas  as  person¬ 
nel  selection,  social  research,  clinical  assessment,  and  anthropological  field 
work. 
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Brocudu tv .  Interviews  vary  in  terms  of  the  amount  oi  st  i  n.imc  mu  t  h*- 
type  of  infonnation  to  be  obtained. 

Structured  interviews  consist  of  a  predetermined,  standardised  set  ot 
questions.  The  questions  may  be  closed,  in  which  the  responses  are  limited 
to  specified  alternatives,  or  open,  in  which  the  respondent  replies  freely. 
Closed  questions  are  more  efficient  in  obtaining  factual  or  easily  quantifi¬ 
able  data,  such  as  age  or  education.  Open-ended  questions  are  more  appropri¬ 
ate  when  the  issue  is  complex,  when  exploring  the  formulation  of  an  opinion, 
or  when  the  relevant  dimensions  of  an  issue  are  not  known. 

Less  structured  interviews  are  commonly  used  for  a  more  intensive  study 
of  the  basic  issues  involved  in  a  topic,  the  terminology,  level  of  understand¬ 
ing,  or  conceptualization  of  a  topic.  If  the  investigator  is  skilled  at  de¬ 
veloping  appropriate  questions  during  the  course  of  the  interview,  highly  spe¬ 
cific  and  self- revealing  data  are  provided  by  the  respondent.  Unstructured 
interviews  might  take  various  forms:  the  nondirective  interview,  the  clinical 
interview,  the  stress  interview,  and  the  group  interview  (Selltiz,  Wrightsman, 
&  Cook,  1976) . 

Benefits  and  Limitations.  The  interview  can  be  used  as  a  data  collection 
method  with  nearly  all  segments  of  the  population,  literate  or  not.  The  in¬ 
vestigator  is  afforded  a  good  deal  of  flexibility  in  terms  of  collecting  data 
about  complex  issues  and  exploring  areas  where  questions  are  difficult  to 
formulate.  In  addition,  the  interview  provides  a  greater  opportunity  for 
the  investigator  to  be  sensitive  to  misunderstanding  of  respondents'  ques¬ 
tions.  However,  the  freedom  which  the  investigator  is  permitted  can  also  be 
a  disadvantage  because  it  creates  problems  in  comparing  one  interview  with 
another.  Also  on  the  negative  side,  interviews  place  heavy  reliance  on  the 
validity  of  verbal  reports.  Even  if  they  are  valid,  verbal  reports  might  be 
misleading,  particularly  regarding  the  causes  and  control  of  one's  own  be¬ 
havior  (Selltiz  et  al.,  1976). 
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In  Chapter  9,  a  detailed  comparison  of  the  merits  and  pitfalls  of  ques¬ 
tionnaires  and  interviews  is  presented.  Question  content  used  to  as¬ 
certain  information  about  facts,  beliefs,  expectations,  and  intentions 
is  discussed.  Types  of  interviews  and  questionnaires  ranging  from 
rigidly  standardized  to  unstructured  are  detailed.  A  brief  section 
deals  with  the  use  of  questionnaires  in  the  field  of  sociometry,  the 
study  of  social  interactions.  Finally,  the  use  of  visual  aids  during 
the  interview  is  discussed. 
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Definition.  A  questionnaire  is  a  written  list  ui  questions  used  i  <  •  ob¬ 
tain  factual  or  subjective  information  about  specified  topics. 

History  and  Applications.  Like  the  interview,  the  questionnaire  has  been 
used  for  a  number  of  years  and  in  a  variety  of  fields,  especially  in  personnel 
and  social  psychology,  consumer  research,  and  attitude  and  opinion  research. 

Procedure.  A  large  number  of  factors  such  as  clarity,  directness,  and 
order  of  presentation  should  be  considered  in  formulating  questions  and  de¬ 
signing  questionnaires.  Selltiz  et  al.  (1976)  and  Bouchard  (1976)  present  a 
good  discussion  of  these  considerations  and  a  number  of  references  for  further 
investigation . 

Benefits  and  Limitations.  Questionnaires  are  more  economical  than  in¬ 
terviews  because  they  can  be  administered  to  a  large  number  of  respondents 
simultaneously.  Furthermore,  little  skill  is  needed  to  administer  question¬ 
naires,  and  more  uniform  measurement  is  possible  with  questionnaires  as  com¬ 
pared  to  interviews,  because  the  data  are  limited  to  written  responses  to 
preset  questions.  In  addition,  the  respondents  may  have  more  confidence  in 
their  anonymity  and  so  respond  more  candidly.  Moreover,  the  respondents  may 
feel  less  pressure  to  respond  immediately  and  so  may  give  more  carefully  con¬ 
sidered  answers. 

Questionnaires  are  limited  by  two  main  factors:  (1)  Only  very  structured 
questionnaires  with  limited  responses  (e.g.,  yes,  no)  can  be  effectively  ad¬ 
ministered  to  the  general  population.  Open-ended  questionnaires  are  useful 
only  for  a  relatively  small,  educated  segment  of  the  population.  (2)  If  the 
questionnaires  are  mailed  rather  than  administered,  a  sizable  proportion  of 
the  respondents  does  not  return  them.  See  Erdos  (1970)  or  Bouchard  (1976) 
for  suggestions  to  increase  the  percentage  of  returns. 
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Uet  lnition.  Scientific  observation  refers  to  fht.  selective  to.,  using  oi 
attention  by  an  investigator  to  obtain  data  in  such  a  manner  that  it  is  pos¬ 
sible  to  summarize,  simplify,  and  systematize  a  behavioral  event  to  be  con¬ 
sistent  with  the  investigator's  research  goals. 

History  and  Applications.  Observation  is  a  pervasive  activity  in  daily 
life,  as  well  as  a  primary  method  of  scientific  inquiry  in  laboratory  and 
field  research.  Observational  methods  are  used  by  all  the  behavioral  sci¬ 
ences,  including  developmental  and  social  psychology,  human  factors,  anthro¬ 
pology,  and  program  evaluation. 

Procedure .  The  relationship  of  the  observer  to  the  observed  varies  along 
two  important  dimensions:  concealment  and  intervention.  Concealment  of  the 
observer's  role  can  be  classified  into  four  categories:  (1)  as  a  complete 
participant,  the  researcher  conceals  the  investigative  role  and  surreptitiously 
becomes  a  group  member;  (2)  as  an  observer,  the  researcher's  role  is  somewhat 
revealed  but  participation  in  group  activities  is  carried  out  to  minimize  dis¬ 
ruption  of  the  situation;  (3)  the  observer  as  participant  is  the  typical  an¬ 
thropologist  role;  the  researcher's  role  is  publicly  known  and  informants  are 
heavily  used  to  provide  data;  and  (4)  the  complete  observer  is  totally  candid 
about  the  investigative  role  and  may  employ  film,  videotape,  or  tape  recorders 
to  gather  data. 

Several  formalized  techniques  have  been  developed  for  recording  on-going 
behavior.  They  are  field  notes,  specimen  records,  anecdotes,  and  checklists. 

Field  notes  are  a  running  log  of  observations.  Ideally,  they  should  con¬ 
tain  information  about:  (1)  the  participants  (e.g.,  how  many,  who  are  they, 
and  how  are  they  related),  (2)  the  setting,  (3)  the  purpose  of  the  situation, 
(4)  the  social  behavior  (e.g.,  what  were  its  qualities,  what  occurred,  and 
what  were  the  initiating  stimuli,  and  (5)  frequency  and  duration  of  the 
situation.  The  recorded  activities  should  be  concrete  and  defined  behavior- 
ally;  however,  the  researcher's  inferences  and  personal  feelings  may  be  in¬ 
cluded,  provided  they  are  labeled  as  such. 

Specimen  records  describe  time-ordered  behavior  in  its  context  and  usu¬ 
ally  over  brief  time  periods.  This  method  is  applied  to  events  with  specific 
temporal  qualities,  such  as  a  music  lesson  or  job  interview.  Very  detailed, 
verbatim  data  are  recorded,  as  well  as  the  researcher's  inferences  and  feel¬ 
ings.  The  records  are  usually  quite  lengthy;  one  study  of  a  single  day  in 
one  boy's  life  produced  420  pages.  With  specimen  records,  an  attempt  is 
made  to  understand  the  behavior  of  one  individual  in  a  specific  setting, 
while  field  notes  are  used  to  understand  behavior  in  a  specific  setting  across 
individuals . 

Anecdotes  are  the  most  widely  used  method  for  describing  behavior  in 
natural  settings.  Unlike  the  two  previous  methods,  the  behaviors  of  inter¬ 
est  are  selected  prior  to  data  collection  and  in  comparison  to  specimen 
records.  Anecdotes  contain  objective  and  quantifiable  data. 

Each  of  the  above  methods  (field  notes,  specimen  records,  and  anecdotes) 
results  in  a  narrative  record  of  on-going  behavior.  What  distinguishes  them 
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ol  the  ub.siuv.U  mi,  period--us  well  as  the  level  of  detail  of  the  recorded 
data.  Because  these  methods  are  relatively  unstructured,  few  restrictions 
are  placed  on  the  type  of  data  collected.  In  addition,  a  unit  of  analysis 
may  not  be  well  defined  at  the  outset.  Therefore,  these  methods  are  very 
susceptible  to  problems  relating  to  validity,  reliability,  observer  bias, 
and  memory  distortions.  For  these  reasons,  unstructured  methods  are  more 
suited  to  hypothesis  generation  than  hypothesis  testing. 

When  a  great  deal  of  information  about  the  situation  of  interest  is  known 
a  more  structured  method  of  collecting  data  is  more  appropriate.  One  such 
method  is  the  checklist.  It  is  used  to  record  whether  or  not  explicitly  speci¬ 
fied,  operationally  defined  behaviors  are  present,  and  results  in  a  frequency 
count  of  those  behaviors.  (Selltiz  et  al.,  1976,  provide  a  good  description 
of  the  above  methods.) 

In  the  examination  of  work  and  equipment  layout,  human  factors  special¬ 
ists  might  draw  from  a  group  of  observational  methods  known  as  process  analy¬ 
sis  techniques.  There  are  four  types  of  such  techniques:  process  charts, 
flow  diagrams,  multiple-process  charts,  and  link  analysis. 

1.  A  process  chart  describes  the  steps  in  a  repetitive  and  standardized 
operation  such  as  an  assembly  line,  through  the  use  of  symbols  which  denote 
transportation,  operation,  storage,  and  inspection.  Transportation  refers  to 
the  movement  from  one  place  to  another  of  the  objects  or  operators  of  inter¬ 
est.  Operation  refers  to  the  main  activity  of  an  operator.  Storage  can  be 
either  temporary  or  controlled  and  refers  to  an  object  being  held  at  one  lo¬ 
cation  without  an  operation  being  performed  on  it.  Inspection  can  be  quanti¬ 
tative,  in  which  an  object  is  counted,  weighed,  or  measured,  or  it  can  be 
qualitative,  in  which  an  object  is  tested  to  determine  whether  or  not  it  meets 
some  predetermined  criterion. 

2.  A  flow  diagram  is  a  graph  of  a  process  chart  showing  the  locations 
of  the  operations,  or  it  may  show  simply  the  paths  of  movement.  For  example, 
the  path  of  eye  movements  required  by  a  pilot  in  the  execution  of  a  preflight 
check-off  can  be  described  by  a  flow  diagram. 

3.  A  multiple-process  chart  is  similar  to  a  process  chart  except  it  is 
concerned  with  multiple  processes,  and  time  values  are  recorded  to  provide 
information  on  how  long  each  step  takes.  For  example,  a  multiple-process 
chart  could  be  applied  to  the  analysis  of  the  activities  of  a  pilot  and  co¬ 
pilot  in  landing  a  plane.  Videotapes  or  multiple  observers  must  be  used  to 
collect  data. 

4.  Link  analysis  refers  to  a  flow  diagram  in  which  the  linkages  between 
component  parts  (humans  and  machines)  are  expressed  statistically.  Link  val¬ 
ues  designate  the  relative  importance  of  each  link.  They  are  obtained  by 
counting  the  number  of  times  a  link  is  used  during  real  or  simulated  opera¬ 
tions  or  by  having  an  experienced  person  assign  the  values.  For  example, 
link  analysis  could  be  applied  to  the  eye  movements  of  a  pilot  during  instru¬ 
ment  landing,  in  order  to  evaluate  the  layout  of  the  instrument  panel. 
(Chapanis,  1959) 


Benefits  and  Limitations.  Observational  methods  are  useful  when  subjects 
cannot  or  will  not  provide  self-report  measures  of  their  behavior.  Observa¬ 
tion  provides  descriptive  data  about  on-goinq  behaviors.  Observation  cannot 
provide  information  about  perceptions,  attitudes,  future  plans,  or  past  or 
private  behavior  (Selltiz  et  al . ,  1976). 

Serious  limitations  to  the  observational  approach ,  especially  when  used 
in  applied  settings,  are  the  following:  (1)  observation  may  affect  the  be¬ 
havior  being  studied;  (2)  significant  relationships  among  factors  that  influ¬ 
ence  behavior  are  difficult  to  determine;  (3)  many  useless  data  may  be  col¬ 
lected  unless  behaviors  of  interest  are  clearly  specified  in  advance;  and 
(4)  some  observations  are  costly  and  time  consuming  (Chapanis ,  1959). 
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Unobtrusive  Measures 


Definition .  Unobtrusive  measures  are  those  in  which  the  investigator 
plays  a  passive,  unobserved  role  or  avoids  contact  with  the  subject  altogether 
by  examining  physical  traces  or  records  of  past  behavior. 

History  and  Applications.  Unobtrusive  research  is  a  very  old  method  and 
is  widely  used  by  police  investigators,  detectives,  archaeologists,  and  social 
researchers . 

Procedure .  Four  major  categories  of  unobtrusive  measures  are  physical 
trades,  archives,  simple  observation,  and  measures  gathered  with  hardware. 
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1.  Physical  traces  produced  by  natural  erosion  or  accretion,  such  as 
worn  tiles,  missing  pages,  or  garbage,  can  be  indices  of  frequency  of  use. 

In  a  study  of  attitudes,  physical  traces  can  be  used  to  determine,  for  ex¬ 
ample,  whether  the  size  of  male  and  female  grave  markers  reflects  the  amount 
of  sexism  in  different  communities,  or  in  a  study  of  decision  making  to  de¬ 
termine,  for  example,  consumer  decisions  regarding  the  disposal  of  various 
kinds  of  material  possessions  through  an  examination  of  garbage.  In  an  addi¬ 
tional  example,  the  nu'nber  of  cigarette  butts  in  an  ashtray  could  be  used  as 
an  indicator  of  the  amount  of  anxiety  generated  in  a  meeting.  Sources  of 
misinterpretation  should  be  carefully  considered  when  studying  physical  traces 
because  this  information  is  a  very  indirect  indicator  of  psychological 
processes . 

2.  Archives  include  both  public  and  private  records  such  as  media  com¬ 
munications;  police,  judicial,  and  institutional  records;  and  personal  com¬ 
munications.  An  example  of  the  use  of  archives  in  a  psycnological  study  is 
the  examination  of  trial  transcripts  or  judicial  records  to  determine  whether 
judges  are  biased  against  members  of  a  certain  ethnic  group  or  socio-economic 
status.  Although  archives  can  be  useful  sources  of  information,  the  investi¬ 
gator  should  be  aware  that  records  are  sometimes  juggled  for  personal  or  po¬ 
litical  reasons. 

3.  Simple  observation  refers  to  situations  in  which  the  researcher  is 

a  disguised  participant  or  is  completely  concealed.  Any  treatment  is  applied 
and  data  are  collected  without  the  subjects'  knowledge.  Simple  observations 
can  provide  information  on  nonverbal  phenomena  like  social  distance,  dominance, 
order  of  participation,  amount  of  interaction,  and  choice  among  alternatives. 

4.  Hardware  such  as  tape  and  video  recorders,  still  and  infrared  pho¬ 
tography,  telemetry,  and  voice  transmitters  provide  an  enormous  potential  for 
data  collection.  (Webb,  Campbell,  Schwartz,  and  Sechrest,  1966,  and  Sechrest, 
1979,  provide  discussions  of  the  above  methods.) 

Benefits  and  Limitations.  Unobtrusive  measures  are  appropriate  when 
overt,  situational  behavior  or  traces  of  past  behavior  can  satisfy  some  re¬ 
search  interest.  The  primary  advantage  of  unobtrusive  research  is  that  the 
behavior  under  study  is  not  contaminated  by  reactivity,  i.e.,  the  subject  is 
unaware  that  she  or  he  is  being  observed  or  tested.  Lack  of  reactivity  does 
not  guarantee,  however,  that  the  results  are  valid,  as  there  are  other  pos¬ 
sible  confounding  sources  such  as  selective  recording  of  archives  or  delib¬ 
erate  original  distortion  of  traces . 

Practical  and  ethical  factors  should  be  considered  when  conducting  un¬ 
obtrusive  research.  Invasion  of  privacy  may  be  an  issue  when  observing  a 
subject  or  examining  personal  records.  In  addition,  certain  records,  espe¬ 
cially  private  records  and  sometimes  police  or  judicial  records,  are  diffi¬ 
cult  to  obtain. 


REFERENCES  AND  SELECTED  ANNOTATIONS 

Bouchard,  T.  J.,  Jr.  (1976).  Field  research  methods.  In  M.  D.  Dunnette 

(Ed.),  Handbook  of  industrial  and  organizational  psychology.  Chicago: 
Rand  McNally,  pp.  363-413.  (See  Questionnaire  References  section  for 
an  annotation  of  this  chapter.) 


24 


Wushi  liqtiin. 


Sechrest,  L.  (Ed.).  (1979).  Unobtrusive  measurement  today . 

D.C.:  Jossey-Bass. 

Webb,  E.  G. ,  Campbell,  D.  T. ,  Schwartz,  R.  D. ,  &  Sechrest,  L.  (i9bb).  Un¬ 
obtrusive  measures:  Nonreactive  research  in  the  social  sciences.  Chi¬ 
cago:  Rand  McNally.  The  book  presents  an  extensive  survey  of  unobtru¬ 

sive  methodologies  employed  to  obtain  social  science  data.  The  authors 
criticize  the  wide  use  of  the  interview  and  the  questionnaire  in  social 
science  research  on  the  basis  that  they  create  as  well  as  measure  atti¬ 
tudes,  they  elicit  atypical  roles,  and  they  are  limited  to  cooperative 
populations.  The  authors  seek  to  present  alternative  and  frequently 
imaginative  approaches  that  avoid  these  biases.  The  approaches  can  be 
used  in  conjunction  with  other  methods  to  cross-validate  results. 

In  Chapter  1,  the  issue  of  multiple  operationism  is  addressed. 

This  refers  to  confirmation  through  multiple  methods,  each  having  dif¬ 
ferent  bias  patterns.  In  Chapter  2,  the  study  of  physical  traces  cre¬ 
ated  by  natural  erosion  or  accretion  is  considered  as  a  method  to  in¬ 
vestigate  past  behavior.  In  Chapters  3  and  4,  the  use  of  archives, 
records  produced  for  purposes  other  than  those  which  are  scholarly  are 
examined.  The  former  focuses  on  continuous  archives  such  as  actuarial 
records;  political,  judicial,  and  governmental  records;  and  media  com¬ 
munications.  In  the  latter,  discontinuous,  private  records  such  as 
sales  records,  industrial  and  institutional  records,  and  personal  docu¬ 
ments  are  examined.  The  final  two  chapters  discuss  observation  of  be¬ 
havior.  In  Chapter  5,  situations  in  which  the  observer  plays  a  passive, 
unobserved  role  are  discussed.  In  Chapter  6,  studies  in  which  the  ob¬ 
server  is  concealed  but  structures  the  situation  by  the  use  of  hardware 
or  a  confederate  are  detailed.  An  extensive  collection  of  references 
concludes  the  book. 


Cognitive  Applications 

Field  methods  have  generally  been  used  in  the  study  of  behavioral  and 
task-related  variables.  However,  observation,  interviews,  and  questionnaires 
can  also  be  used  to  explore  cognition  and  decision  making  with  only  a  shift 
in  the  focus  from  overt  to  covert  variables.  Interviews,  with  question  de¬ 
sign  based  on  observed  performance  and  laboratory  research,  could  be  con¬ 
ducted  to  explicate  the  subjective  process  leading  to  a  decision  or  judgment 
(e.g. ,  medical  diagnosis).  For  further  investigation,  questionnaires  could 
be  derived  from  interview  data  and  administered  on  a  larger  scale.  The  re¬ 
sults  would  provide  insight  into  information  processing,  perceptual,  and 
decision-making  issues.  As  with  the  traditional  use  of  these  methods,  how¬ 
ever,  the  results  would  be  colored  by  subject  bias  and  recall  limitations. 

Unobtrusive  observation  can  be  used  to  study  actual  decision-making  be¬ 
havior  in  the  real  world.  For  example,  Ebbesen  and  Konecni  (1978)  investi¬ 
gated  judges'  bail-setting  decisions  experimentally  (in  simulations,  and 
unobtrusively)  in  real  court  hearings.  Ebbesen,  Parker,  and  Konecni  (1977) 
examined  drivers'  intersection-crossing  decisions  in  laboratory  simulations 
and  at  real  intersections.  In  both  of  these  studies,  unobtrusive  observa¬ 
tion  proved  not  only  useful,  but  crucial,  to  the  proper  interpretation  and 
validation  of  laboratory  findings. 


The  increasingly  widespread  use  ol  computers  it.  bn.-.  1  ness  setting.-,  should 
enable  researchers  to  validly  and  unobtrusively  measure  cognitive  processes . 
For  example,  questions  pertaining  to  cognitive  biases  could  be  disguised  and 
incorporated  in  computer  software.  The  operator  would  answer  then,  while  pro¬ 
ceeding  through  a  sequence  of  iob-related  computer  procedures.  Similarly, 
questions  could  be  asked  of  students  using  computer-aided  instruction.  Com¬ 
pared  with  a  job-related  sequence  of  questions,  questions  incorporated  in 
computer-aided  instruction  software  would  probably  run  little  risk  of  arous- 
inq  the  operator's  suspicion  since  the  procedure  is  less  apt  to  be  routine. 
Therefore,  embedding  and  concealing  questions  should  be  more  easily 
accompl ished . 
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CONCLUSIONS 

1.  Nonlaboratory  methods  for  the  study  of  cognition  and  decision  pro¬ 
cesses  do  exist.  Methods  found  to  be  useful  are  (1)  verbal  protocol  analy¬ 
sis,  (2)  information-acquisition  behavior ,  (3)  critical  incident  technique, 

(4)  Delphi  method,  (5)  interviews,  (6)  questionnaires,  (7)  observation,  and 
(8)  unobtrusive  measures. 

No  procedure  will  ever  allow  direct  assessment  of  covert  cognitive  pro¬ 
cesses.  Inferences  from  behavior  will  always  be  required.  Of  the  eight 
methods,  the  first  two  (categorized  as  process-tracing  methods)  were  designed 
specifically  for  the  study  of  cognition.  They  require  little  adaptation  to 
be  used  in  nonlaboratory  settings  and  can  provide  much  specific  information 
about  memory,  problem  solving,  decision  making,  and  perception.' 

The  next  four  methods  (critical  incident,  Delphi,  interview,  and  ques¬ 
tionnaire)  rely  heavily  on  retrospection  and  self-report.  Because  of  the 
likelihood  of  inherent  biases  such  as  recall  limitations,  these  methods  can 
be  better  used  for  collectinq  data  to  generate  rather  than  test  hypotheses. 

Observation  of  behavior  and  unobtrusive  measurement  of  artifacts,  ar¬ 
chives,  and  on-going  behavior  in  natural  settings  are  also  valid  methods  to 
assess  cognition.  However,  because  of  the  possible  need  for  a  larger  number 
of  assumptions  to  infer  cognitive  processing,  and  the  potential  for  collect¬ 
ing  voluminous  amounts  of  data,  these  two  methods  also  might  be  better  for 
hypothesis  generation. 

Operations  research,  the  ninth  method  explored  in  this  report ,  does  not 
provide  a  way  to  assess  cognitive  processing.  It  could  be  useful,  however, 
for  developing  models,  and  for  testing  and  predicting  the  implications  of 
such  processing. 
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2.  Most  nonlaboratory  methods  will  tequi  r<  reativt  adaptation  betorf- 
they  can  be  employed  in  research.  A  prerequisite  for  any  effort  toward  adap¬ 
tation  is  a  determination  of  which  coqnitive  processes  (e.g.,  perception, 
learning)  are  required  in  the  performance  of  a  particular  task.  Therefore, 
both  a  background  knowledge  in  human  cognition  as  wel)  as  a  thorough  under¬ 
standing  of  the  task  are  necessary. 

3.  The  computer  environment  will  provide  a  rich  and  flexible  context 
for  applying  nonlaboratory  techniques  and  methods.  Examples  are:  (1)  proto¬ 
cols  and  information-acquisition  strategies  can  be  unobtrusively  collected 

by  examining  the  interactive  dialogue  between  the  computer  and  user;  and 
(2)  data  collection  (questionnaire,  interview  information,  information  usage) 
can  be  easily,  continuously,  and  perhaps  surreptitiously  accomplished  by 
monitoring  user  activity  with  the  computer. 
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